Skip to content

fix(server): preserve accepted automation runs during recovery - #164

Merged
ScriptedAlchemy merged 1 commit into
mainfrom
fix/deep-automation-recovery-audit-2026-11-18
Nov 19, 2026
Merged

ScriptedAlchemy merged 1 commit into
mainfrom
fix/deep-automation-recovery-audit-2026-11-18

Conversation

@ScriptedAlchemy

Copy link
Copy Markdown
Owner

Two failure paths could lose ownership of work already accepted by the orchestrator:

  • A coalesced webhook could commit its command, fail to record completion, then lose its duplicate-delivery protection when recovery encountered a newly paused task. Redelivery could start the work again.
  • A launch or bound-message send could commit successfully before a reply read failed. Manual tasks, one-time scheduled tasks, and immediate/coalesced webhooks then recorded an incorrect failure. Webhooks could release their replay claims; other tasks could lose their returned thread, preparation recovery, or terminal-outcome tracking.

The scheduler now reconciles a failed reply with the exact canonical command receipt. Accepted work proceeds through completion tracking with its original command and message IDs. An accepted launch whose run is still preparing resumes through the existing idempotent preparation service, using the durable message and run instead of rendering or dispatching another message.

Pending webhook recovery checks acceptance before pause/trigger guards and payload preparation. It preserves the original delivery log and restores an eligible lost first-thread binding while respecting task identity, project, current mode, and an intervening user edit. Missing/rejected receipts retain ordinary failure behavior. A failed receipt lookup retains duplicate protection and reports uncertainty without claiming that the run started.

Proof

The regressions extend existing tests:

  • The original paused-recovery defect failed with one failing row and three controls.
  • Independent review added seven recovery cases covering accepted/missing/rejected receipts, pause, lost first-thread binding, a later switch to new-thread mode, and a framing failure after acceptance. The two hardening failures were reproduced before correction.
  • The real-service integration table has sixteen cases: four original attribution controls and three acceptance/reply failure boundaries for each of manual, due-once, immediate webhook, and coalesced webhook execution. The six webhook cases and then the six manual/once cases failed before their respective corrections.
  • These cases use real SQLite/EventSink commits and the actual thread/preparation services. They verify one durable message/run, successful occurrence bookkeeping, retained attribution and prompt, and settlement out of preparing. Due-once cases also verify a single occurrence count and no remaining next-run time.

The before-preparation fixture commits the real thread-create command and initial message, then injects a failure at the service boundary. It does not intercept an actual projection-read implementation or claim live-provider execution.

Final validation

On the clean publication tree based on reviewed main 09efb428b74ef6a1d22be25bdba9c62f8a180669, vp run check:changed 09efb428b74ef6a1d22be25bdba9c62f8a180669 --concurrency 1 passed native build, formatting, lint, both full Knip scans, GraphQL generation, and 186 tests across six files, including the task, webhook, launch, and budget checks.

The cached compiler launcher cannot initialize seccomp_unotify here, so the wrapper exits nonzero. All three affected packages passed their configured effect-tsgo tsrs --noEmit checks directly: server, evals, and scripts. Compiler errors found during development were corrected without casts, suppressions, or weakened checks. Independent review covered receipt identity, interruption, preparation, binding, all four entry points, and the existing failure finalizer.

The published Git tree is checked equal to the locally validated tree. GitHub Fork CI including final Fork Check, Agent Context Budget, and every other applicable check must pass on this exact head before merge.

Recovery boundary

Coalesced deliveries have durable pending rows that can revisit uncertain acceptance. Immediate deliveries retain their existing service-scoped lifetime. If both an immediate dispatch reply and its receipt lookup fail, the replay claim is retained; this PR does not add automatic durable reconciliation for that double-failure case. Retrying safely requires resolving the canonical receipt first. Private/live providers and native platform execution are not claimed.

Implemented and verified with GPT-6 Astra Pro / ChatGPT.

Reconcile task command receipts after failed dispatch replies. Recover accepted preparation and eligible bindings, retain webhook replay ownership across completion failures, and cover manual, scheduled, immediate, and coalesced delivery paths.
@changeset-bot

changeset-bot Bot commented Nov 19, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 4d9c3e2

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@ScriptedAlchemy ScriptedAlchemy left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact head 4d9c3e2. The final isolated tree passed 186 tests across six files, native build, formatting, lint, both Knip scans, GraphQL checks, and direct configured server/evals/scripts compilers. The cached compiler wrapper remains blocked by this environment's seccomp_unotify restriction; no gate was weakened.

Recovery boundary: immediate webhook admission has no durable pending row. If both the dispatch reply and receipt lookup fail, this change retains duplicate protection while acceptance is unknown. Coalesced rows can retry recovery; automatic durable reconciliation for immediate delivery is not established here. Resolve the canonical receipt before any retry or claim release. A broader durability change would need persisted immediate-delivery intent plus restart/fault-injection proof.

The accepted-before-preparation tests commit through the real EventSink and exercise the existing preparation service. Private/live-provider execution is not claimed. All applicable GitHub checks, including final Fork Check and Agent Context Budget, remain required before merge.

@ScriptedAlchemy
ScriptedAlchemy merged commit 26d72fd into main Nov 19, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant